Systems and metohds of gesture recognition

ABSTRACT

Disclosed methods include a method of controlling a computing device includes the steps of detecting a gesture made by a human user, identifying the gesture, and executing a computer command. The gesture may comprise a change in depth of a body part of the human user relative to the 2D camera. The gesture may be detected via a 2D camera in electronic communication with the computing device. Disclosed systems include a 2D camera and a computing device in electronic communication therewith. The 2D camera is configured to capture at least a first and second image of a body part of a human user. The computing device is configured to recognize at least a first object in the first image and a second object in the second image, identify a change in depth, and execute a command in response to the change in depth.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority, to U.S.application Ser. No. 14/796,944, filed on Jul. 10, 2015, which is herebyincorporated by reference for all purposes.

BACKGROUND

The present disclosure relates generally to systems and devices forgesture recognition. In particular, systems and devices usingtwo-dimensional (2D) cameras that are capable of gesture recognition inthree-dimensional (3D) space are described.

Gesture recognition is a rapidly growing technological field forinputting commands to computer systems, gaming systems, and/or otherinteractive control systems. In such systems, human gestures arecaptured by image and/or motion detection subsystems and interpreted viaalgorithms to relay specific commands to the subject system. Gesturescan originate from any bodily motion or state, but commonly originatefrom the face or hand of a user.

Known gesture recognition devices and systems are not entirelysatisfactory for the range of applications in which they are employed.For example, existing gesture recognition systems and devices havedifficulty recognizing gestures that are performed in 3D space. Morespecifically, existing gesture recognition systems have difficultyrecognizing gestures that require “depth perception”, such as a forwardswipe gesture (i.e., hand moving away from the device) and/or a rearwardswipe gesture (i.e., hand moving toward the device). In another example,some gesture recognition systems and devices include 3D cameras (e.g.,structure light cameras, time-of-flight cameras, stereo cameras, etc.)to track gestures through 3D space. These systems, however, areexpensive and require complicated systems for operation.

Thus, there exists a need for gesture recognition devices and systemsthat improve upon and advance the design of known gesture recognitiondevices and systems. Examples of new and useful gesture recognitiondevices and systems relevant to the needs existing in the field arediscussed below.

SUMMARY

In one embodiment, a method of controlling a computing device includesthe steps of detecting a gesture made by a human user, identifying thegesture, and executing a computer command. The gesture may comprise achange in depth of a body part of the human user relative to the 2Dcamera. The gesture may be detected via a 2D camera in electroniccommunication with the computing device. The detecting step may includecapturing at least a first image of the body part in an initial gestureposition and a second image of the body part in a subsequent gestureposition. The gesture may be identified via a computing unit of thecomputing device. The identifying step may comprise recognizing at leasta first object in the first image and a second object in the secondimage, comparing at least the first object to the second object, andconverting the gesture into a command via the computing unit. Theexecuting step may be triggered in response to the gesture beingconverted into a command.

In one embodiment, a system includes a 2D camera and a computing device.The computing device may be in electronic communication with the 2Dcamera. In the embodiment, the 2D camera is configured to capture atleast a first and second image of a body part of a human user. In theembodiment, the computing device is configured to recognize at least afirst object in the first image and a second object in the second image.In the embodiment, the computing device is configured to generate afirst boundary in the first image, the first boundary being the smallestregion of the first image that fully contains the first object. In theembodiment, the computing device is configured to generate a secondboundary in the second image, the second boundary being the smallestregion of the second image that fully contains the second object. In theembodiment, the computing device is configured to compare at least thesize of the first boundary to the size of the second boundary in orderto identify a change in depth of the body part of the human userrelative to the 2D camera. In the embodiment, the computing device isconfigured to convert the change in depth of the body part into acommand. In the embodiment, the computing device is configured toexecute the command.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic view of an example of a programmable computingdevice.

FIG. 2 shows a schematic view of an example of a mobile electronicdevice.

FIG. 3 is a schematic view of a first example of a gesture recognitionsystem.

FIGS. 4A and 4B are schematic views of a second example of a gesturerecognition system and a third example of the gesture recognitionsystem, respectively.

FIGS. 5A and 5B are example camera views of example hand gestures fromany of the first, second, or third example gesture recognition systems.

FIGS. 6A-6C are example camera views of overlaid hand gesture movementimages from any of the first, second, or third example gesturerecognition systems.

DETAILED DESCRIPTION

The disclosed WIDGETS will become better understood through review ofthe following detailed description in conjunction with the figures. Thedetailed description and figures provide merely examples of the variousinventions described herein. Those skilled in the art will understandthat the disclosed examples may be varied, modified, and altered withoutdeparting from the scope of the inventions described herein. Manyvariations are contemplated for different applications and designconsiderations; however, for the sake of brevity, each and everycontemplated variation is not individually described in the followingdetailed description.

Throughout the following detailed description, examples of variousWIDGETS are provided. Related features in the examples may be identical,similar, or dissimilar in different examples. For the sake of brevity,related features will not be redundantly explained in each example.Instead, the use of related feature names will cue the reader that thefeature with a related feature name may be similar to the relatedfeature in an example explained previously. Features specific to a givenexample will be described in that particular example. The reader shouldunderstand that a given feature need not be the same or similar to thespecific portrayal of a related feature in any given figure or example.

The disclosed gesture recognition systems and devices using 2D camerasfor detecting gestures through 3D space will become better understoodthrough review of the following detailed description in conjunction withthe figures. The detailed description and figures provide merelyexamples of the various inventions described herein. Those skilled inthe art will understand that the disclosed examples may be varied,modified, and altered without departing from the scope of the inventionsdescribed herein. Many variations are contemplated for differentapplications and design considerations; however, for the sake ofbrevity, each and every contemplated variation is not individuallydescribed in the following detailed description.

Throughout the following detailed description, a variety of gesturerecognition systems and devices using 2D cameras for detecting gesturesthrough 3D space examples are provided. Related features in the examplesmay be identical, similar, or dissimilar in different examples. For thesake of brevity, related features will not be redundantly explained ineach example. Instead, the use of related feature names will cue thereader that the feature with a related feature name may be similar tothe related feature in an example explained previously. Featuresspecific to a given example will be described in that particularexample. The reader should understand that a given feature need not bethe same or similar to the specific portrayal of a related feature inany given figure or example.

With reference to FIGS. 1-5 examples of gesture recognition systemsusing 2D cameras to detect gestures made through 3D space, gesturerecognition systems 300, 400 and 500, will now be described. Thepresently described gesture recognition systems and devices function todetect and interpret gesture commands performed through 3D space (e.g.,a forward swipe gesture, a rearward swipe gesture, etc.). Additionallyor alternatively, the presently described gesture recognition systemsand devices can function to detect and interpret generally 2D gesturecommands (i.e., gestures made in a generally flat plane).

The presently described gesture recognition systems and devices addressmany of the shortcomings existing with conventional gesture recognitionsystems and devices. For example, using the gesture recognition systemsof the present application, a 2D camera can be used to recognize depthand detect gestures that are made in 3D space by comparing a firstboundary of a first image (i.e., an image of a beginning of a gesture)to a second boundary of a second, subsequent, image (i.e., an image ofan ending of the gesture). Further, the presently described gesturerecognition systems and devices do not require of 3D camera, thereforethey have an overall reduced cost and complexity.

As described below, in some embodiments, the gesture recognition systemsof the present application may be employed to control or interact withone or more computing devices. In this regard, it will be appreciatedthat various disclosed examples may be implemented using electroniccircuitry configured to perform one or more functions. For example, withsome embodiments of the invention, the disclosed examples may beimplemented using one or more application-specific integrated circuits(ASICs). More typically, however, components of various examples of theinvention will be implemented using a programmable computing deviceexecuting firmware or software instructions, or by some combination ofpurpose-specific electronic circuitry and firmware or softwareinstructions executing on a programmable computing device.

Accordingly, FIG. 1 shows one illustrative example of a computingdevice, computing device 101, which can be used to implement variousembodiments of the invention. Computing device 101 may be incorporatedwithin a variety of consumer electronic devices, such as personal mediaplayers, cellular phones, smart phones, personal data assistants, globalpositioning system devices, smart eyewear, smart watches, other computerwearables, and the like.

As seen in this figure, computing device 101 has a computing unit 103.Computing unit 103 typically includes a processing unit 105 and a systemmemory 107. Processing unit 105 may be any type of processing device forexecuting software instructions, but will conventionally be amicroprocessor device. System memory 107 may include both a read-onlymemory (ROM) 109 and a random access memory (RAM) 111. As will beappreciated by those of ordinary skill in the art, both read-only memory(ROM) 109 and random access memory (RAM) 111 may store softwareinstructions to be executed by processing unit 105.

Processing unit 105 and system memory 107 are connected, either directlyor indirectly, through a bus 113 or alternate communication structure toone or more peripheral devices. For example, processing unit 105 orsystem memory 107 may be directly or indirectly connected to additionalmemory storage, such as a hard disk drive 117, a removable optical diskdrive 119, a removable magnetic disk drive 125, and a flash memory card127. Processing unit 105 and system memory 107 also may be directly orindirectly connected to one or more input devices 121 and one or moreoutput devices 123.

Output devices 123 may include, for example, a monitor display, anintegrated display, television, printer, stereo, or speakers. Inputdevices 121 may include, for example, a keyboard, touch screen, a remotecontrol pad, a pointing device (such as a mouse, touchpad, stylus,trackball, or joystick), a scanner, a microphone, or a camera.

More specifically, in the presently described gesture recognitionsystems, input devices 121 include at least a 2D camera 122 (e.g., alight camera, a thermographic camera, etc.). In one example, 2D camera122 is a visible light digital camera. The visible light digital camerauses an optical system including a lens and a variable diaphragm tofocus light onto an electronic image pickup device. The visible lightdigital camera can be a compact digital camera, a bridge camera, amirrorless interchangeable-lens camera, a modular camera, a digitalsingle-lens reflex camera, digital single-lens translucent camera,line-scan camera, etc. Further, it will be appreciated that the visiblelight digital camera can be any known or yet to be discovered visiblelight digital camera.

In one embodiment, 2D camera 122 is integral to the computing device103. In another embodiment, 2D camera 122 is remote of the computingdevice 103.

As mentioned above, 2D camera 122 can additionally or alternatively be athermographic camera or infrared (IR) camera. The IR camera can detectheat radiation in a way similar to the way an ordinary camera detectsvisible light. This makes IR cameras useful for gesture recognition in“normal light”, “low light”, and/or “no light” conditions. The IR cameracan include cooled infrared photodetectors (e.g. indium antimonide,indium arsenide, mercury cadmium telluride, lead sulfide, lead selenide,etc.) and/or uncooled infrared photodetectors (e.g., vanadium oxide,lanthanum barium manganite, amorphous silicon, lead zirconate titanate,lanthanum doped lead zirconate titanate, lead scandium tantalate, leanlanthanum titanate, lead titanate, lead zinc niobate, lead strontiumtitanate, barium strontium titanate, antimony sulfoiodide,polyvinylidene difluoride, etc.). Further, it will be appreciated thatthe IR camera can be any known or yet to be discovered thermographiccamera.

Returning to FIG. 1, computing unit 103 can be directly or indirectlyconnected to one or more network interfaces 115 for communicating with anetwork. This type of network interface 115, also sometimes referred toas a network adapter or network interface card (NIC), translates dataand control signals from computing unit 103 into network messagesaccording to one or more communication protocols, such as theTransmission Control Protocol (TCP), the Internet Protocol (IP), and theUser Datagram Protocol (UDP). These protocols are well known in the art,and thus will not be discussed here in more detail. An interface 115 mayemploy any suitable connection agent for connecting to a network,including, for example, a wireless transceiver, a power line adapter, amodem, or an Ethernet connection.

It should be appreciated that, in addition to the input, output andstorage peripheral devices specifically listed above, the computingdevice may be connected to a variety of other peripheral devices,including some that may perform input, output and storage functions, orsome combination thereof. For example, the computing device 101 may beconnected to a digital music player, such as an IPOD® brand digitalmusic player or iOS or Android based smartphone. As known in the art,this type of digital music player can serve as both an output device fora computing device (e.g., outputting music from a sound file or picturesfrom an image file) and a storage device.

In addition to a digital music player, computing device 101 may beconnected to or otherwise include one or more other peripheral devices.In one example, computing device 101 may be connected to or otherwiseinclude a telephone. The telephone may be, for example, a wireless“smart phone,” such as those featuring the Android or iOS operatingsystems. As known in the art, this type of telephone communicatesthrough a wireless network using radio frequency transmissions. Inaddition to simple communication functionality; a “smart phone” may alsoprovide a user with one or more data management functions, such assending, receiving and viewing electronic messages (e.g., electronicmail messages, SMS text messages, etc.), recording or playing back soundfiles, recording or playing back image files (e.g., still picture ormoving video image files), viewing and editing files with text (e.g.,Microsoft Word or Excel files, or Adobe Acrobat files), etc. Because ofthe data management capability of this type of telephone, a user mayconnect the telephone with computing device 101 so that their datamaintained may be synchronized.

In another example, computing device 101 may be connected to orotherwise include an eyewear article (such as eyewear article 302 shownin FIG. 3). The eyewear article may be, for example, a “smart” eyeweararticle, such as a wearable computing device like Google® Glass.

The “smart” eyewear technologies are particularly suited to the displayof “augmented reality” displays. “Augmented reality” displays comprise acomputer generated graphical display laid over a portion of a user's ormobile device's natural field of vision. These “augmented reality”displays allow a user to view computer generated images including datarelated to objects in their natural field of vision. Augmented realitydisplays may include any display including both natural and computergenerated elements.

Of course, still other peripheral devices may be included with orotherwise connected to a computing device 101 of the type illustrated inFIG. 1, as is well known in the art. In some cases, a peripheral devicemay be permanently or semi-permanently connected to computing unit 103.For example, with many computing devices, computing unit 103, hard diskdrive 117, removable optical disk drive 119 and a display aresemi-permanently encased in a single housing.

Still other peripheral devices may be removably connected to computingdevice 101, however. Computing device 101 may include, for example, oneor more communication ports through which a peripheral device can beconnected to computing unit 103 (either directly or indirectly throughbus 113). These communication ports may thus include a parallel bus portor a serial bus port, such as a serial bus port using the UniversalSerial Bus (USB) standard or the IEEE 1394 High Speed Serial Busstandard (e.g., a Firewire port). Alternately or additionally, computingdevice 101 may include a wireless data “port,” such as a Bluetooth®interface, a Wi-Fi interface, an infrared data port, or the like.

It will be appreciated that a computing device employed accordingvarious examples of the invention may include more components thancomputing device 101 illustrated in FIG. 1, fewer components thancomputing device 101, or a different combination of components thancomputing device 101. Some implementations of the invention, forexample, may employ one or more computing devices that are intended tohave a very specific functionality, such as a digital music player orserver computer. These computing devices may thus omit unnecessaryperipherals, such as the network interface 115, removable optical diskdrive 119, printers, scanners, external hard drives, etc. Someimplementations of the invention may alternately or additionally employcomputing devices that are intended to be capable of a wide variety offunctions, such as a desktop or laptop personal computer. Thesecomputing devices may have any combination of peripheral devices oradditional components as desired.

In many examples, computing devices may comprise mobile electronicdevices, such as smart phones, smart glasses, tablet computers, orportable music players, often operating the iOS, Symbian, Windows-based(including Windows Mobile and Windows 8), or Android operating systems.

With reference to FIG. 2, an exemplary computing device, mobile device200, may include a processor unit 203 (e.g., CPU) configured to executeinstructions and to carry out operations associated with the mobiledevice. For example, using instructions retrieved from memory, thecontroller may control the reception and manipulation of input andoutput data between components of the mobile device. The controller canbe implemented on a single chip, multiple chips or multiple electricalcomponents. For example, various architectures can be used for thecontroller, including dedicated or embedded processor, single purposeprocessor, controller, ASIC, etc. By way of example, the controller mayinclude microprocessors, DSP, A/D converters, D/A converters,compression, decompression, etc.

In most cases, the controller together with an operating system operatesto execute computer code and produce and use data. The operating systemmay correspond to well-known operating systems such iOS, Symbian,Windows-based (including Windows Mobile and Windows 8), or Androidoperating systems, or alternatively to special purpose operating system,such as those used for limited purpose appliance-type devices. Theoperating system, other computer code and data may reside within asystem memory 207 that is operatively coupled to the controller. Systemmemory 207 generally provides a place to store computer code and datathat are used by the mobile device. By way of example, system memory 207may include read-only memory (ROM) 209, random-access memory (RAM) 211.Further, system memory 207 may retrieve data from storage units 294,which may include a hard disk drive, flash memory, etc. In conjunctionwith system memory 207, storage units 294 may include a removablestorage device such as an optical disc player that receives and playsDVDs, or card slots for receiving mediums such as memory cards (ormemory sticks).

Mobile device 200 also includes input devices 221 that are operativelycoupled to processor unit 203. Input devices 221 are configured totransfer data from the outside world into mobile device 200. As shown,input devices 221 may correspond to both data entry mechanisms and datacapture mechanisms. In particular, input devices 221 may include touchsensing devices 232 such as touch screens, touch pads and touch sensingsurfaces, mechanical actuators 234 such as button or wheels or holdswitches, motion sensing devices 236 such as accelerometers, locationdetecting devices 238 such as global positioning satellite receivers,WiFi based location detection functionality, or cellular radio basedlocation detection functionality, force sensing devices 240 such asforce sensitive displays and housings, image sensors 242 such as lightcameras and/or IR cameras, and microphones 244. Input devices 221 mayalso include a clickable display actuator.

More specifically, in the presently described gesture recognitionsystems, input devices 221 include at least a 2D camera 243 (one ofimage sensing devices 242). 2D camera 243 can be a visible light cameraand/or a thermographic camera, such as those described above inreference to 2D camera 122. Accordingly, 2D camera 243 has the samefunctions and capabilities as those described above in reference to 2Dcamera 122.

Returning to FIG. 2, mobile device 200 also includes various outputdevices 223 that are operatively coupled to processor unit 203. Outputdevices 233 are configured to transfer data from mobile device 200 tothe outside world. Output devices 233 may include a display unit 292such as an LCD, speakers or jacks, audio/tactile feedback devices, lightindicators, and the like.

Mobile device 200 also includes various communication devices 246 thatare operatively coupled to the controller. Communication devices 246may, for example, include both an I/O connection 247 that may be wiredor wirelessly connected to selected devices such as through IR, USB, orFirewire protocols, a global positioning satellite receiver 248, and aradio receiver 250 which may be configured to communicate over wirelessphone and data connections. Communication devices 246 may also include anetwork interface 252 configured to communicate with a computer networkthrough various means which may include wireless connectivity to a localwireless network, a wireless data connection to a cellular data network,a wired connection to a local or wide area computer network, or othersuitable means for transmitting data over a computer network.

Mobile device 200 also includes a battery 254 and possibly a chargingsystem. Battery 254 may be charged through a transformer and power cordor through a host device or through a docking station. In the cases ofthe docking station, the charging may be transmitted through electricalports or possibly through an inductance charging means that does notrequire a physical electrical connection to be made.

The various aspects, features, embodiments or implementations describedabove can be used alone or in various combinations with the gesturerecognition methods disclosed herein. The methods disclosed herein canbe implemented by software, hardware or a combination of hardware andsoftware. The methods can also be embodied as computer readable code ona computer readable medium (e.g. a non-transitory computerreadable-storage medium). The computer readable medium is any datastorage device that can store data which can thereafter be read by acomputer system, including both transfer and non-transfer devices asdefined above. Examples of the computer readable medium includeread-only memory, random access memory, CD-ROMs, flash memory cards,DVDs, magnetic tape, optical data storage devices, and carrier waves.The computer readable medium can also be distributed overnetwork-coupled computer systems so that the computer readable code isstored and executed in a distributed fashion.

Turning now to FIGS. 3-4B example gesture recognition systems are shownand described. As depicted in FIG. 3, a first example of a gesturerecognition system, gesture recognition system 300, is shown implementedan eyewear article 302. Eyewear article 302 is a “smart” eyeweararticle, such as those described above in reference to FIGS. 1 and 2.Accordingly, eyewear article 302 includes a frame 304 for supporting acomputing device 306 (such as computing device 101) including at least aprocessor 308 and memory 310 in data communication with an imagegenerator 312 and a 2D camera 314. Image generator is configured todisplay and/or project an image into the view of a user wearing eyeweararticle 302, creating an augmented reality view.

2D camera 314 can be any of the cameras described above in reference to2D camera 122. 2D camera 314 is configured to detect a gesture 316 thatchanges in depth relative to the camera (i.e., moves away from thecamera or moves toward the camera). In the present example, gesture 316is an open hand gesture; however, it will be appreciated that 2D camera314 can detect a variety of other hand gestures (e.g., closed hand, openhand, swipe right, swipe left, swipe forward, swipe backward, etc.)and/or a variety of gestures (e.g., wink of eye, blink of eyes, headmovements up, down, right, and/or left, etc.). A gesture may comprise amovement from an initial gesture position to a subsequent gesturepositions. As used herein, a gesture position is a particularorientation of a body part of a person. It will be further appreciatedthat gesture 316 and/or any other gesture detected by 2D camera 314 canbe a gesture made by a user (i.e., a wearer of the eye wear article) ora gesture made by another person.

A second example of a gesture recognition system, gesture recognitionsystem 400, is shown in FIG. 4A and will now be described. Gesturerecognition system 400 includes many similar or identical features togesture recognition system 300. Thus, for the sake of brevity, eachfeature of gesture recognition system 400 will not be redundantlyexplained. Rather, key distinctions between gesture recognition system400 and gesture recognition system 300 will be described in detail andthe reader should reference the discussion above for featuressubstantially similar between the gesture recognition systems.

As can be seen in FIG. 4A, gesture recognition system 400 is implementedin a mobile computing device 402 including a 2D camera 414 in datacommunication with a computing unit (not specifically shown) of themobile device for detecting a gesture 416. In the present example,mobile device 402 is a smart phone. It will be appreciated that in otherexamples mobile device 402 can be any of the mobile devices describedabove in reference to mobile device 200 (shown in FIG. 2).

2D camera 414 can be any of the cameras described above in reference to2D camera 122. 2D camera 414 is configured to detect a gesture 416 thatchanges in depth relative to the camera (i.e., moves away from thecamera or moves toward the camera). In the present example, gesture 416is an open hand gesture; however, it will be appreciated that 2D camera414 can detect a variety of other hand gestures (e.g., closed hand, openhand, swipe right, swipe left, swipe forward, swipe backward, etc.)and/or a variety of gestures (e.g., wink of eye, blink of eyes, headmovements up, down, right, and/or left, etc.).

A third example of a gesture recognition system, gesture recognitionsystem 500, is shown in FIG. 4B and will now be described. Gesturerecognition system 500 includes many similar or identical features togesture recognition systems 300 and 400. Thus, for the sake of brevity,each feature of gesture recognition system 500 will not be redundantlyexplained. Rather, key distinctions between gesture recognition system500 and gesture recognition systems 300 and 400 will be described indetail and the reader should reference the discussion above for featuressubstantially similar between the gesture recognition systems.

As can be seen in FIG. 4B, gesture recognition system 500 is implementedin a computing device 502 including a 2D camera 514 in datacommunication with a processor and memory of the computing device (notspecifically shown) for detecting a gesture 516. In the present example,computer device 502 is a laptop computer. It will be appreciated that inother examples computing device 502 can be any of the mobile devicesdescribed above in reference to computing device 101 (shown in FIG. 1).

2D camera 514 can be any of the cameras described above in reference to2D camera 122. 2D camera 514 is configured to detect a gesture 516 thatchanges in depth relative to the camera (i.e., moves away from thecamera or moves toward the camera). In the present example, gesture 516is an open hand gesture; however, it will be appreciated that 2D camera514 can detect a variety of other hand gestures (e.g., closed hand, openhand, swipe right, swipe left, swipe forward, swipe backward, etc.)and/or a variety of gestures (e.g., wink of eye, blink of eyes, headmovements up, down, right, and/or left, etc.).

Turning now to FIGS. 5A-6B, example camera views and boundaries asdetected and calculated by a camera in the above described systems(e.g., cameras 122, 243, 314, 414, and 514). Specifically, FIG. 5A showscamera view 600 of a first example gesture made by a user, closed handgesture 616 (i.e., a fist). Closed hand gesture 616 is recognized andidentified as a “closed hand command” by a system in data communicationwith the camera. In one specific example, a “closed hand command” is acommand to select/grab an image on a screen and/or a virtual image. Inanother specific example, a “closed hand command” is a command tocapture an image. In even another specific example, a “closed handcommand” is a command to select and/or move forward through a series ofscreens and/or modes. Further, movement of gesture 616 is tracked by thesystem via the collected camera images.

In the example of camera view 600, a first boundary 618 is generatedand/or calculated when the user's hand is in an initial gestureposition. A second boundary 620 is generated when the user's hand is asubsequent gesture position.

Thus, in this example, gesture 616 is tracked as it moves away from thecamera. In other words, gesture 616 is a rearward moving gesture. Themovement of gesture 616 is recognized and identified by the system and acommand is sent to the image generator to execute the command. In onespecific example, a rearward movement triggers a command to decrease asize of a virtual image in an augmented reality view (i.e., a “zoom outcommand”). In other embodiments, a rearward movement may trigger acommand to adjust the volume up, adjust the volume down, scroll up,scroll down, turn on, turn off, open an application, or close anapplication, among others.

FIG. 5B depicts a camera view of a second example gesture made by auser, an open hand gesture 716 (i.e., an open palm). Open hand gesture716 is recognized and identified as an “open hand command” by a systemin data communication with the camera. In one specific example, a “openhand command” is a command to select/grab an image on a screen and/or avirtual image. In another specific example, an “open hand command” is acommand to select and/or move backward through a series of screensand/or modes. Further, movement of gesture 716 is tracked by the systemvia the collected camera images.

In the example of camera view 700, a first boundary 718 is generatedand/or calculated when the user's hand is in an initial gestureposition. A second boundary 720 is generated when the user's hand is afirst subsequent gesture position, and a third boundary 722 is generatedwhen a user's hand is in a second subsequent gesture position (i.e., aposition after the first subsequent gesture position).

Thus, in this example, gesture 716 is tracked as it toward the camera.In other words, gesture 716 is a forward moving gesture. The movement ofgesture 716 is recognized and identified by the system and a command issent to the image generator to execute the command. In one specificexample, a forward movement is a command to increase a size of a virtualimage in an augmented reality view (i.e., a “zoom in command”). It willbe appreciated that a forward movement can be associated with adifferent command (i.e., a turn off command, a turn on command, etc.).

FIGS. 6A-6C, show overlaid successive camera views 800, 900, and 1000,respectively. Each of these views shows a closed hand gesture movingaway from the camera (i.e., a rearward moving gesture). In the exampleof FIG. 6A, three successive boundaries are calculated, boundaries 818,820, and 822 for a closed hand gesture 816. In the example of FIG. 6B,two successive boundaries, boundaries 918 and 920, are calculated forclosed hand gesture 916. In the example of FIG. 6C, three successiveboundaries, boundaries 1018, 1020, and 1022, are calculated for a closedhand gesture 1016. In the example of FIG. 6C, gesture 1018 is a rearwardand rightward moving gesture. In this example, the rearward andrightward moving gesture can include an additional or alternativecommand. For example, in an augmented reality display a rearward andrightward gesture can be a “zoom out” and “move right” command formovement of a virtual image.

It will be appreciated that different hand signs (e.g. open hand, closedhand, etc.) can be recognized and identified as different objects by asystem in data communication with a camera. The change in depth (e.g.,rearward movement, forward movement, etc.) is recognized when the samehand sign is detected in successive camera frames with a changing size.Thus, in one embodiment, the computing device 103 is configured toidentify a first object in a first image and second object in a secondimage. The first object may be indicative of an initial gestureposition. For example, the first object may be a hand sign, or a portionof a hand sign. The second object may be indicative of a subsequentgesture position. For example, the second object may be the same handsign, or portion thereof, however the second object may be a differentsize than the first object.

As soon as a different hand sign/gesture is detected, the depth gesturethat was in progress can be considered completed. Thus, in someembodiments, after identifying a first initial gesture position, andthen identifying a second initial gesture position, the computing device103 is configured to terminate any command associated with the firstinitial gesture position.

A boundary (e.g., boundaries 616, 618, 718, 720, 722, 818, 820, 822,918, 920, 1018, 1020, and/or 1022) can be calculated using an objectdetection algorithm. Suitable object detection algorithms include theHaar Cascade Classifier algorithm that finds objects in an image thatclosely match a given set of image features, and the HOG detectorclassifier. The search is done by comparing regions of the full imageagainst the features being searched for. The smallest region that fullycontains the matching object is considered the object's boundary.Subsequent boundaries can be generated over time, creating two or morebounding regions. In some embodiments, the regions have predeterminedshapes. For example the regions may be specified as rectangles, circles,triangles, and trapezoids, among others.

Depth is calculated by comparing the relative widths and/or heights ofthe two or more bounding regions. In some instances, relative width orheight is used depending on whether the hand position being used for thegesture is bigger horizontally or vertically. When a subsequent boundingregion is bigger in whichever dimension is being used for the comparison(e.g., height, width, circumference, and/or area) than an initialbounding region, the hand is considered to have moved closer to thecamera (i.e., a forward moving gesture). When a subsequent boundingregion is smaller than the initial bounding region, the hand isconsidered to have moved farther from the camera (i.e., a rearwardmoving gesture).

The magnitude of the depth change may be quantified, via computing unit103, by scaling the difference in width and/or height by an application-and/or device-specific constant. Thus, the magnitude of the change indepth may be quantified. In this regard, a magnitude may be associatedwith a gesture triggered command. For example, a gesture comprising achange in depth of 2 ft. may be converted, via computing unit 103, intoa command to decrease the size of an image on a display unit (i.e.,“zoom out”) by a factor of 200%. In another example, a gesturecomprising a change in depth of 1 ft. may be converted, via computingunit 103, into a command to decrease the size of an image on a displayunit by a factor of 100%. As can be appreciated, a magnitude may beassociated with any of the commands discussed above.

Optionally, the scaling constant can also change dynamically based onthe speed of the gesture. For example, the scaling constant can bechanged dynamically by dividing the magnitude by the time differencebetween the two or more bounding regions, which would cause the depthchange magnitude to be bigger when the gesture is performed at a fasterspeed.

The disclosure above encompasses multiple distinct inventions withindependent utility. While each of these inventions has been disclosedin a particular form, the specific embodiments disclosed and illustratedabove are not to be considered in a limiting sense as numerousvariations are possible. The subject matter of the inventions includesall novel and non-obvious combinations and subcombinations of thevarious elements, features, functions and/or properties disclosed aboveand inherent to those skilled in the art pertaining to such inventions.Where the disclosure or subsequently filed claims recite “a” element, “afirst” element, or any such equivalent term, the disclosure or claimsshould be understood to incorporate one or more such elements, neitherrequiring nor excluding two or more such elements.

Applicant(s) reserves the right to submit claims directed tocombinations and subcombinations of the disclosed inventions that arebelieved to be novel and non-obvious. Inventions embodied in othercombinations and subcombinations of features, functions, elements and/orproperties may be claimed through amendment of those claims orpresentation of new claims in the present application or in a relatedapplication. Such amended or new claims, whether they are directed tothe same invention or a different invention and whether they aredifferent, broader, narrower or equal in scope to the original claims,are to be considered within the subject matter of the inventionsdescribed herein.

What is claimed is:
 1. A method of controlling a computing device, themethod comprising: detecting, via a 2D camera in electroniccommunication with the computing device, a gesture made by a human userof the computing device; wherein the gesture comprises a change in depthof a body part of the human user relative to the 2D camera wherein depthis a dimension that is measured parallel to a line extendingperpendicularly from a lens of the 2D camera; and wherein the detectingstep comprises capturing at least a first image of the body part in aninitial gesture position at an initial depth and a second image of thebody part in a subsequent gesture position at a subsequent depth,wherein the subsequent depth is different than the initial depth;identifying the gesture via a computing unit of the computing device,wherein the identifying step comprises: recognizing at least a firstobject in the body part of the first image and a second object in thebody part of the second image, wherein the recognizing step comprises;generating a first boundary in the first image that fully contains thefirst object, wherein the first object corresponds to the body part inthe initial gesture position; generating a second boundary in the secondimage that fully contains the second object, wherein the second objectcorresponds to the body part in the subsequent gesture position;determining the change in a magnitude depth from the initial gestureposition to the subsequent gesture position without requiringdetermination of an initial magnitude depth or a subsequent magnitudedepth, wherein the determining comprises comparing a size of the firstboundary to a size of the second boundary; converting the gesture into acommand via the computing unit; and executing, in response to theconverting step, the command via the computing unit.
 2. The method ofclaim 1, wherein the recognizing step includes using an object detectionalgorithm to recognize the first and second images.
 3. The method ofclaim 1, wherein the first boundary is the smallest region of the firstimage that fully contains the body part of the human user, and whereinthe second boundary is the smallest region of the second image thatfully contains the body part of the human user.
 4. The method of claim3, wherein the object detection algorithm is selected from the groupconsisting of a Haar Cascade Classifier and a HOG Detector Classifier.5. The method of claim 1, wherein: when the first object is smaller thanthe second object, the gesture is identified as a forward movinggesture; and when the first object is larger than the second object, thegesture is identified as a rearward moving gesture.
 6. The method ofclaim 5, comprising converting a rearward moving gesture into a commandto decrease a size of an image on a display unit in electroniccommunication with the computing device.
 7. The method of claim 5,comprising converting a foreword moving gesture into a command toincrease a size of an image on a display unit in electroniccommunication with the computing device.
 8. The method of claim 1,wherein the comparing step includes quantifying a difference in at leastone of the width of the first boundary and the second boundary and theheight of the first boundary and the second boundary using a scalingconstant; and
 9. The method of claim 8, wherein the scaling constant ischanged dynamically by dividing the change in magnitude depth by thetime difference between the first boundary and second boundary.
 10. Themethod of claim 8, wherein the converting step includes generating amagnitude associated with the command, wherein the magnitude is based onthe quantified difference in the at least one of the width of the firstboundary and the second boundary and the height of the first boundaryand the second boundary
 11. The method of claim 10, wherein thecomparing step includes analyzing a speed of the gesture, and whereinthe converting step includes adjusting the magnitude associated with thecommand based on the speed of the gesture.
 12. The method of claim 1,wherein the initial gesture position comprises a particular orientationof the body part of the human user.
 13. The method of claim 12, whereinthe initial gesture position is one of an open hand and a closed hand.14. A system comprising: a 2D camera configured to capture at least afirst and second image of a gesture made by a human user, wherein thegesture comprises a change in depth of a body part of the user relativeto the 2D camera, and wherein depth is a dimension that is measuredparallel to a line extending perpendicularly from a lens of the 2Dcamera; a computing device in electronic communication with the 2Dcamera configured to: recognize at least a first object in the body partof the first image and a second object in the body part of the secondimage, wherein the first object corresponds to the body part in aninitial gesture position at an initial depth and the second objectcorresponds to the body part in a subsequent gesture position at asubsequent depth, and wherein the subsequent depth is different than theinitial depth; generate a first boundary in the first image, the firstboundary being the a smallest region of the first image that fullycontains the first object; generate a second boundary in the secondimage, the second boundary being the a smallest region of the secondimage that fully contains the second object; compare at least a size ofthe first boundary to a size of the second boundary in order todetermine a magnitude of the change in depth of the body part of thehuman user relative to the 2D camera without requiring determination ofan absolute magnitude of the initial depth or an absolute magnitude ofthe subsequent depth; convert the change in depth of the body part intoa command; and execute the command.
 15. The system of claim 14, whereinthe computing device is configured to recognize the first and secondobjects via an object detection algorithm.
 16. The system of claim 14,wherein the computing device is configured to identify a first gesturein a first initial gesture position.
 17. The system of claim 16, whereinthe computing device is configured to identify a second gesture in asecond initial gesture position and terminate the first command inresponse to the identification of the second gesture.
 18. The system ofclaim 14, wherein the 2D camera is integral to the computing device. 19.The system of claim 14, wherein the 2D camera is remote of the computingdevice.
 20. The system of claim 14, comprising a display unit inelectronic communication with the computing device configured to displaya virtual image to the human user.